PStream: A Popularity-Aware Differentiated Distributed Stream Processing System

Real-world stream data with skewed distributions raises unique challenges to distributed processing systems. Existing workload partitioning schemes usually use a “one size fits all” design, which leverages either shuffle grouping or key strategy for the workloads among multiple units, leading notable problems of unsatisfied system throughput and latency. In this article, we show that based result in serious load imbalance low computation efficiency presence skewness while are not scalable terms memory space. We argue efficient scheduling is popularity data. propose PStream, popularity-aware differentiated assigns hot keys using rare ones grouping. PStream novel light-weighted probabilistic counting scheme identifying currently dynamic real-time streams. The extremely consumption, so predictor on it can be well integrated into instances system. further design an adaptive threshold configuration scheme, quickly adapt dynamical changes highly implement top Apache Storm conduct comprehensive experiments large-scale traces from real-world systems evaluate performance design. Results achieves 2.3× improvement reduces latency by 64 percent compared state-of-the-art designs.